1. Field
This disclosure relates to data stored in a data storage system and an improved architecture and method for storing data to and retrieving data from a data storage system particularly in a high speed super computing environment.
2. Description of the Related Art
A file system is used to store and organize computer data stored as electronic files. File systems allow files to be found, read, deleted, and otherwise accessed. File systems store files on one or more storage devices. File systems store files on storage media such as hard disk drives and solid-state storage devices.
Various applications may store large numbers of documents, images, audio, videos and other data as objects using a distributed data storage system in which data is stored in multiple locations.
Parallel log-structured file system techniques were introduced in the Zest checkpointing file system and the Parallel Log-Structured File system (PLFS). Both Zest and PLFS allow clients to operate in a fully autonomous fashion not subject to coherency management mechanisms. Other parallel file systems such as PanFS® (available from Panasas, Inc.), Lustre® (available from the Cluster File Systems, Inc.), and GPFS (the General Parallel File System available from IBM Corp.) use page based schemes which require network level update atomicity. In these file systems, clients acquire a lock before updating a file or portion thereof, otherwise, the system risks corrupting files when two or more clients attempt to update the same page simultaneously. Zest and PLFS provide internal structures which allow for the indexing of file extents at single byte granularity.
When large amounts of data are created quickly, Zest and PLFS have significant deficiencies when a data item must be accessed. In a Zest system, data may only be obtained from primary storage, typically a hard disk drive. Faster, buffer volatile memory may not be accessed to obtain requested data. In Zest, the system incurs delays while waiting for data to be written to primary before being accessed. In PLFS, data stored in buffer memory may be accessed, but it is indexed linearly such that significant time delays are incurred in identifying the location of data to be read from a buffer in PLFS systems.
Another file system, PanFS®, the parallel file system product from Panasas, Inc., institutes a network RAID scheme that builds fault tolerant groups from aligned contiguous file regions. This approach suffers from serialization in that only one client may modify a given region at any one time. Further, this scheme is susceptible to a large degree of read-modify-writes which causes performance degradation.
Throughout this description, elements appearing in figures are assigned three-digit reference designators, where the most significant digit is the figure number and the two least significant digits are specific to the element. An element that is not described in conjunction with a figure may be presumed to have the same characteristics and function as a previously-described element having a reference designator with the same least significant digits.